System for engagement of human agents for decision-making in a dynamically changing environment

ABSTRACT

Disclosed is a system and a method for engagement of human agents for decision-making in a dynamically changing environment. An information request related to a problem requiring a decision is received. Further, problem data comprising metadata associated to the problem, and decision-making data is received. Then, an information type is determined for the information request. Subsequently a set of human agents from a list of one or more human agents is determined using an engagement model. Further, a request elicitation type is determined for the set of human agents using an elicitation model. Further, an input is received from the set of human agents. Further, the input is used to retrain the engagement model and the elicitation model. Finally, the decision-making data is continuously enhanced based on the input received, the request elicitation type, and the information type.

PRIORITY INFORMATION

The present application is a continuation-in-part of prior U.S.application Ser. No. 17/643,635, filed Dec. 10, 2021.

TECHNICAL FIELD

The present subject matter described herein, in general, relates to asystem and a method for utilizing automation in decision making, andmore particularly to utilization of automation in man-machinecollaborative decision making in a dynamically changing environment.

BACKGROUND

Decision making is a complex process that involves analysis of the data.Making an informed decision has been simplified with increaseddigitization and improvement in computer technology. The internet hasplayed an important role in making the data readily available foreveryone to use. Machines can now execute jobs that were previously onlypossible for humans, such as making decisions, thanks to technologicaladvancements and the introduction of artificial intelligence and deeplearning. The decision making is relevant to any field of humanendeavour some examples may be resource management, service management,government, commercial industry, asset management, security management,and a host of other fields that require decision making. There arevarious systems available for assisting humans in decision making asprocessing data from a repository is not a difficult task for a machine.The various systems available today have one common problem. The worldis dynamic, and most of the processes we come across in our daily livesare non-linear and depend on dynamic, interdependent parameters, whichmakes it difficult for the various systems to comprehend real-lifesituations and arrive at a conclusion.

SUMMARY

Before the present system(s) and method(s), are described, it is to beunderstood that this application is not limited to the particularsystem(s), and methodologies described, as there can be multiplepossible embodiments which are not expressly illustrated in the presentdisclosures. It is also to be understood that the terminology used inthe description is for the purpose of describing the particularimplementations or versions or embodiments only and is not intended tolimit the scope of the present application. This summary is provided tointroduce aspects related to a system and a method for engagement ofhuman agents for decision-making in dynamically changing environments.This summary is not intended to identify essential features of theclaimed subject matter nor is it intended for use in determining orlimiting the scope of the claimed subject matter.

In one implementation, a system for engagement of human agents fordecision-making in dynamically changing environments. The system mayreceive an information request relating to a problem requiring adecision. Further, problem data corresponding to the problem may bereceived. It may be noted that the problem data may comprise metadataassociated to the problem, and decision-making data. The metadataassociated to the problem may comprise at least a goal, constraints,success measures, a list of the one or more human agents involved in adecision-making process, and historic data comprising informationreceived from the one or more human agents, a success ratio of theinformation received from the one or more human agents. Thedecision-making data may comprise at least one or more intermediatesteps, importance of the one or more intermediate steps, adecision-making flow, and historic data comprising past informationrequests received. Further, the system may determine an information typebased on the problem data. The information type may be at least a fact,an opinion, and a judgement. It may be noted that the system may use anacquisition model to determine the information type.

Further, the system may determine a set of human agents, for theinformation request, from the list of one or more human agents based onthe problem data. The set of human agents may be determined using anengagement model. It may be noted that the set of human agents maycomprise one or more subsets of human agents for each intermediate stepof the one or more intermediate steps from the decision-making data.Subsequently, the system may determine a Request Elicitation Type (RET)for the set of human agents based on the problem data and the determinedinformation type. The RET may be determined using an elicitation model.It may be noted that the RET may correspond to how the informationrequest is framed for a particular human agent. The system may thenreceive an input from the set of human agents for the informationrequest based on the information type, and the RET. The input may be atleast a text response, a visual response, an audio response, a videoresponse, and a feedback. It may be noted that the system may use theinput received to retrain the engagement model using recursive learningtechniques. Finally, the decision-making data may be continuouslyenhanced based on the input received, the determined request elicitationtype, and the determined information type.

In yet another implementation, non-transitory computer readable mediumembodying a program executable in a computing device for engagement ofhuman agents for decision-making in dynamically changing environments isdisclosed. The program may comprise a program code for receiving aninformation request relating to a problem requiring a decision. Further,the program may comprise a program code for receiving problem datacomprising metadata associated to the problem, and decision-making data.Further, the program may comprise a program code for determining aninformation type based on the problem data. The information type may bedetermined using an acquisition model. It may be noted that theinformation type may be at least a fact, an opinion, and a judgement.Subsequently, the program may comprise a program code for determining aset of human agents from a list of one or more human agents for theinformation request based on the problem data. The set of human agentsmay be determined by using an engagement model. Further, the program maycomprise a program code for determining a Request Elicitation Type (RET)for the set of human agents based on the problem data and theinformation type using an elicitation model. The RET may correspond tohow the information request is framed for a particular human agent.Further, the program may comprise a program code for receiving an inputfrom the set of human agents for the information request based on theinformation type, and the request elicitation type. Furthermore, theprogram may comprise a program code retraining the engagement modelbased on the received input using recursive learning techniques.Finally, the program may comprise a program code for continuouslyenhancing the decision-making data based on the input received, thedetermined request elicitation type, and the determined informationtype.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing detailed description of embodiments is better understoodwhen read in conjunction with the appended drawings. For the purpose ofillustrating of the present subject matter, an example of a constructionof the present subject matter is provided as figures, however, theinvention is not limited to the specific method and system forengagement of human agents in decision making in a dynamically changingenvironment disclosed in the document and the figures.

The present subject matter is described in detail with reference to theaccompanying figures. In the figures, the left-most digit(s) of areference number identifies the figure in which the reference numberfirst appears. The same numbers are used throughout the drawings torefer to various features of the present subject matter.

FIG. 1 illustrates a network implementation of a system for engagementof human agents for decision-making in a dynamically changingenvironment, in accordance with an embodiment of the present subjectmatter.

FIG. 2 illustrates an example of a communication between a human agentand a collaborative decision-making system using an engagement engine,in accordance with an embodiment of the present subject matter.

FIG. 3 illustrates a method for engagement of human agents fordecision-making in a dynamically changing environment, in accordancewith an embodiment of the present subject matter.

FIG. 4 illustrates a method for collaborative decision making in adynamically changing environment, in accordance with an embodiment ofthe present subject matter.

FIG. 5 illustrates an example artificial neural network that may be usedto train a machine learning algorithm, in accordance with an embodimentof the present subject matter.

The figure depicts an embodiment of the present disclosure for purposesof illustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the disclosure described herein.

DETAILED DESCRIPTION

Some embodiments of this disclosure, illustrating all its features, willnow be discussed in detail. The words “receiving,” “extracting,”“determining,” “retraining,” “enhancing,” “modifying,” “calculating,”“generating,” and other forms thereof, are intended to be open ended inthat an item or items following any one of these words is not meant tobe an exhaustive listing of such item or items or meant to be limited toonly the listed item or items. It must also be noted that as used hereinand in the appended claims, the singular forms “a,” “an,” and “the”include plural references unless the context clearly dictates otherwise.Although any system and methods similar or equivalent to those describedherein can be used in the practice or testing of embodiments of thepresent disclosure, the exemplary, system and methods are now described.

The disclosed embodiments are merely examples of the disclosure, whichmay be embodied in various forms. Various modifications to theembodiment will be readily apparent to those skilled in the art and thegeneric principles herein may be applied to other embodiments. However,one of ordinary skill in the art will readily recognize that the presentdisclosure is not intended to be limited to the embodiments describedbut is to be accorded the widest scope consistent with the principlesand features described herein.

The present subject matter discloses a system and a method forengagement of human agents for decision-making in a dynamically changingenvironment. It may be noted that the present subject matter mainlyfocuses on complex decision scenarios with high stakes. The complexdecision scenarios require a great deal of deliberation because of therisk and reward associated with consequences of decision-implementingaction. The complexity of the act of decision-making emanates fromvarious dimensions of the decision-making problem, such as:

The sheer number of factors to be considered in the decision;

Multiple and conflicting goals;

Multiple human agents, each pushing their own perspectives;

Long duration of decision-making and decision implementing actions,during which many factors can change;

Ambiguity and uncertainty of available information;

Human factors—such as differing viewpoints, levels of skills andexperience of decision contributors, emotions, and the effects ofpolitical and other social relationships; and

High risk of not obtaining the desired outcome, and of unintendedconsequences.

The system and the method disclose strategically and systematicallyinvolving concerned human agents for decision making in a high risk,dynamically changing environment. Initially, an information requestrelating to a problem requiring a decision may be received. Further,problem data comprising metadata associated to the problem, anddecision-making data may be received. Subsequently, an information typemay be determined based on the problem data. The information type may bedetermined using an acquisition model. Then, a set of human agents froma list of one or more human agents may be determined based on theproblem data using an engagement model. It may be noted that the set ofhuman agents may be selected based on a participation value, aninformation value, and a human involvement cost. Further, a RequestElicitation Type (RET) for the set of human agents may be determinedbased on the problem data, and the information type using an elicitationmodel. Based on the RET, and the information type, information may bereceived from the set of human agents. The information received may befurther used to retrain the engagement model and the elicitation modelusing recursive learning techniques. Finally, the decision-making datamay be continuously enhanced based on the information received, thedetermined request elicitation type, and the determined informationtype.

Certain technical challenges exist in engagement of human agents fordecision making in a dynamically changing environment. One technicalchallenge faced while determining an information type is that aninformation request may not have a specific requirement or a mention ofthe type of information required to satisfy the information request. Itmay be confusing to answer an information request without knowledge ofwhat is exactly required. The solution presented in the embodimentsdisclosed herein to address the above challenge is machine learningbased acquisition model trained to analyse the problem data, and thedecision-making data and determine what type of information may berequired for a particular information request.

Another technical challenge faced may be determining number of humanagents and which human agents from a list of one or more human agentsmay be required for a particular information request. The involvement ofcorrect human agents and sufficient number of human agents may becrucial for efficient decision-making. The solution presented in theembodiments disclosed herein to address the above challenge is machinelearning based engagement model. The engagement model is trained toanalyse data comprising roles and seniority of the one or more humanagents, historic data about input provided for one or more pastinformation requests, and success ratio of the input provided tocalculate an information value, participation value, and a humaninvolvement cost. Further, the engagement model is continuously trainedto determine a set of human agents from the list of one or more humanagents based on the information value, the participation value, and thehuman involvement cost of each human agent from the list of one or morehuman agents. It may be noted that the engagement model is trained usinghistoric data and the historic data is continuously modified by additionof inputs provided by the one or more human agents for each informationrequest. The inputs provided by the one or more human agents maycomprise a feedback regarding the output of the engagement model. Thefeedback is then used to retrain the engagement model after modifyingthe training data based on the feedback.

Referring now to FIG. 1 , a network implementation 100 of a system 102for engagement of human agents for decision-making in a dynamicallychanging environment is disclosed. Initially, the system 102 receives aninformation request, relating to a problem. It may be noted that adecision may be required for the problem. In an example, the softwaremay be installed on a user device 104-1. It may be noted that the one ormore users may access the system 102 through one or more user devices104-2, 104-3 . . . 104-N, collectively referred to as user devices 104,hereinafter, or applications residing on the user devices 104. Further,the system 102 may also receive feedback from a user using the userdevices 104.

Although the present disclosure is explained considering that the system102 is implemented on a server, it may be understood that the system 102may be implemented in a variety of computing systems, such as a laptopcomputer, a desktop computer, a notebook, a workstation, a virtualenvironment, a mainframe computer, a server, a network server, acloud-based computing environment. It will be understood that the system102 may be accessed by multiple users through one or more user devices104-1, 104-2 . . . 104-N. In one implementation, the system 102 maycomprise the cloud-based computing environment in which the user mayoperate individual computing systems configured to execute remotelylocated applications. Examples of the user devices 104 may include, butare not limited to, a portable computer, a personal digital assistant, ahandheld device, and a workstation. The user devices 104 arecommunicatively coupled to the system 102 through a network 106.

In one implementation, the network 106 may be a wireless network, awired network, or a combination thereof. The network 106 can beimplemented as one of the different types of networks, such as intranet,local area network (LAN), wide area network (WAN), the internet, and thelike. The network 106 may either be a dedicated network or a sharednetwork. The shared network represents an association of the differenttypes of networks that use a variety of protocols, for example,Hypertext Transfer Protocol (HTTP), Transmission ControlProtocol/Internet Protocol (TCP/IP), Wireless Application Protocol(WAP), and the like, to communicate with one another. Further thenetwork 106 may include a variety of network devices, including routers,bridges, servers, computing devices, storage devices, and the like.

In one embodiment, the system 102 may include at least one processor108, an input/output (I/O) interface 110, and a memory 112. The at leastone processor 108 may be implemented as one or more microprocessors,microcomputers, microcontrollers, digital signal processors, CentralProcessing Units (CPUs), state machines, logic circuitries, and/or anydevices that manipulate signals based on operational instructions. Amongother capabilities, the at least one processor 108 is configured tofetch and execute computer-readable instructions stored in the memory112.

The I/O interface 110 may include a variety of software and hardwareinterfaces, for example, a web interface, a graphical user interface,and the like. The I/O interface 110 may allow the system 102 to interactwith the user directly or through the client devices 104. Further, theI/O interface 110 may enable the system 102 to communicate with othercomputing devices, such as web servers and external data servers (notshown). The I/O interface 110 can facilitate multiple communicationswithin a wide variety of networks and protocol types, including wirednetworks, for example, LAN, cable, etc., and wireless networks, such asWLAN, cellular, or satellite. The I/O interface 110 may include one ormore ports for connecting a number of devices to one another or toanother server.

The memory 112 may include any computer-readable medium or computerprogram product known in the art including, for example, volatilememory, such as static random-access memory (SRAM) and dynamicrandom-access memory (DRAM), and/or non-volatile memory, such as readonly memory (ROM), erasable programmable ROM, flash memories, harddisks, Solid State Disks (SSD), optical disks, and magnetic tapes. Thememory 112 may include routines, programs, objects, components, datastructures, etc., which perform particular tasks or implement particularabstract data types. The memory 112 may include programs or codedinstructions that supplement applications and functions of the system102. In one embodiment, the memory 112, amongst other things, serves asa repository for storing data processed, received, and generated by oneor more of the programs or the coded instructions.

As there are various challenges observed in the existing art, thechallenges necessitate the need to build the system 102 for engagementof human agents for decision-making in a dynamically changingenvironment. At first, a user may use the user device 104 to access thesystem 102 via the I/O interface 110. The user may register the userdevices 104 using the I/O interface 110 in order to use the system 102.In one aspect, the user may access the I/O interface 110 of the system102. The detail functioning of the system 102 is described below withthe help of figures.

The present subject matter describes the system 102 for engagement ofhuman agents for decision making in a dynamically changing environment.The system may also be referred to as a Collaborative Decision-makingSystem (CDS). Initially, the system may receive a query from a user. Itmay be noted that the query corresponds to a problem requiring adecision. In an embodiment, the query may be “Expand business into a newterritory,” or “Buy a house,” or “Find a cure for COVID,” or “Make acareer move” or alike.

Further to receiving the query, the system 102 may calculate one or moreintermediate steps required to reach a decision based on metadataassociated to the problem. It may be noted that the one or moreintermediate steps are calculated using reinforcement learning, deeplearning algorithms and artificial intelligence techniques. In anembodiment, the one or more intermediate steps may also be calculatedusing combination of the reinforcement learning, deep learningalgorithms, mathematical modelling, and artificial intelligencetechniques. The metadata may comprise a goal, constraints, successmeasures, a list of the one or more human agents involved in acollaborative decision making. The list of the one or more human agentsmay define the roles and seniority of each of the one or more humanagents. The goal may represent the objective or final expectation of theuser. The constraints may represent the limitations of resourcesavailable to achieve the goal. The success measures may represent theparameters used for confirmation of achievement of the goal.

The one or more human agents may be assigned below roles:

Decision-Maker—A person having an authority to make a decision. Theauthority may be essential to allocate resources to implement thedecision. There may be multiple decision-makers involved in a decisionprogram.

Stakeholder—A person affected by a decision. There may be multiplestakeholders in a decision program bringing in diverse perspectives.

Expert—A person having knowledge relevant to the generation of adecision space. The expert may assess the characteristics of a decisionpoint in the space.

Decision Program Manager—An individual responsible for decision programlife-cycle operations.

Decision Intelligence Manager—An individual responsible for configuringand setting up the system.

System Administrator—An individual responsible for the operationalmanagement of the system.

Subsequent to calculating the one or more intermediate steps, the system102 may establish a decision-making flow for the one or moreintermediate steps required to reach the decision. The decision-makingflow may correspond to a sequence for execution of the one or moreintermediate steps. The sequence may be determined based on animportance and a severity of the one or more intermediate steps usingreinforcement learning and artificial intelligence techniques. Theimportance and the severity of the one or more intermediate steps may bedetermined based on impact of each intermediate step on the decision.The impact of each intermediate step may be calculated using predictivealgorithms. In an embodiment, a decision of an intermediate step may bereferred as an intermediate decision.

In an embodiment, let us assume that one or more intermediate steps areA, B, C, D, and E. Further, the system may determine a sequence forexecution of the one or more intermediate steps. Let us assume that theorder may be “C, B, D A, and E.” It may be noted that the order may bedetermined based on an importance and a severity of the one or moreintermediate steps.

Further to establishing the decision-making flow, the system 102 maygenerate a decision space comprising one or more decision options. Theone or more decision options may be calculated based on thedecision-making flow using artificial intelligence and reinforcementlearning techniques. In an embodiment, the decision space may be emptyinitially. It may be noted that the decision space may represent acriteria to limit the one or more decision options. The decision spacemay be modified by changing the criteria during the decision-makingflow. The decision space may be modified at least by the one or morehuman agents or when an uncertain event occurs. The decision space maybe filled dynamically, during the decision-making flow, with thedecision options that fit the criteria.

A decision option may represent a possible choice for decision contentof a decision in a decision implementing intention. It may be noted thatthe decision-space may be a set whose members are decision options. Themembers may be added or removed from the set during the act ofdecision-making as new information is obtained. A snapshot of thedecision space at a point in time contains all members underconsideration at that time. A decision space model may be a compactrepresentation of the decision space when it is impractical toexplicitly list all members in the space. When needed, members of thedecision space can be generated by executing a generation procedureassociated with the decision-space model.

Further to generating the decision space, the system 102 may modify thedecision space based on one or more uncertain events. The modificationmay indicate addition of one or more new decision options or removal ofone or more decision options. The one or more new decision options maybe calculated based on the new information or uncertain events.Similarly, the removal of the one or more decision options may be basedon the new information or uncertain events. The one or more uncertainevents may create an uncertain impact on the decision-making flow. Theuncertain impact may be calculated using predictive algorithms andartificial intelligence techniques. In an embodiment, the one or moreuncertain events may be eliminated when the uncertain impact is lessthan a predefined threshold.

It may be noted that the one or more uncertain events may lead to a riskhaving consequences. The one or more risk may be calculated usingArtificial Intelligence Techniques. The consequences may be predictedand calculated using predictive algorithms and artificial intelligencetechniques. Consider an example, a company wants to hire employees (aproblem). The one or more risk associated with the problem may be atleast the company received zero applications, zero applicants qualifiedfor the interview, an applicant rejected an offer letter, incompetentcandidate got hired, applicant quits after being hired, and alike. Itmay be noted that the system calculates the consequence for each risk.In an embodiment, the system may suggest preventive measures or riskinformed strategy to reduce the impact of the consequences. The risk maybe low, medium, or high based on the impact of the consequences. Inanother embodiment, the impact of the consequences may be a short term,medium term, or a long term. In yet another embodiment, the systemassists the one or more human agents to select an intermediate decisionto avoid risk.

The one or more uncertain events may be determined based on newinformation. The system may receive new information from external datasources and the input from the one or more human agents at the one ormore intermediate step based on the importance and the severity of thestep. The external data source may comprise internet search, monitoringchanges in published data repositories, subscription to market researchservices, and analysis of news and social media content. In anembodiment, the input from the one or more human agents may be ajudgement, an additional decision option, or alike. The new informationmay be used to calculate the one or more uncertain events or executeconsequent intermediate steps in the decision-making flow. In anembodiment, the system 102 may nudge a human agent from the one or morehuman agents for the input based on a predefined matrix of roles andseniority of the human agent.

In an embodiment, the system may comprise an engagement engine toreceive the input for the new information. The engagement engine mayalso be referred to as the system for engagement of human agents with acollaborative decision-making system or the system for engagement ofhuman agents for decision-making in a dynamically changing environment.

The engagement engine may be configured to execute a set of instructionsfor engagement of human agents with a collaborative decision-makingsystem. In an embodiment, the engagement engine may receive aninformation request relating to a problem that may requiredecision-making. The information request may be received from thecollaborative decision-making system used by a user to reach a decisionregarding the problem. The information request may be received when theCollaborative Decision-making System (CDS) may need at least anapproval, new information, and a feedback from a human agent to proceedwith a decision-making process.

Further to receiving the information request, the engagement engine mayreceive problem data. In an embodiment, the engagement engine mayreceive the problem data from the CDS. The engagement engine may accessa database of the CDS to extract the problem data that may be stored inthe memory or on a server. The problem data may comprise metadataassociated to the problem, and decision-making data. The metadata maycomprise the goal, constraints, success measures, the list of the one ormore human agents involved in a decision-making process, and historicdata comprising input received from the one or more human agents forpast information requests, and a success ratio of the input receivedfrom the one or more human agents. The list of the one or more humanagents may include responsibilities and seniority of each of the one ormore human agents. Further, the decision-making data may comprise theone or more intermediate steps, importance of the one or moreintermediate steps, the decision-making flow, and historic datacomprising past information requests. The one or more intermediate stepsmay be a part of the decision-making process of the collaborativedecision-making system for the problem.

In an embodiment, sending an information request to the engagementengine may be an intermediate step of the decision-making process of theCDS. The decision-making flow may define a sequence of the one or moreintermediate steps. The historic data may comprise past informationrequests, one or more human agents or a set of human agents involved forthe past information requests, and the input received from the involvedset of human agents.

Further to receiving the problem data, the engagement engine maydetermine an information type for the information request based on theproblem data. The information type may be at least a fact, an opinion,and a judgement. The engagement engine may use an acquisition model todetermine the information type. The acquisition model is a machinelearning based model which is dynamically trained using a training datacomprising a plurality of information requests, corresponding problemdata for the plurality of information requests, and correspondinginformation types for each of the plurality of information requests. Theacquisition model may be trained to produce an information type as anoutput for an input of an information request and corresponding problemdata of the information request. In an embodiment, the engagement enginemay configure NLP (Natural Language Processing) algorithms and NLU(Natural Language Understanding) algorithms to synthesize theinformation request.

Consider an example, let us assume that the information request is“Validate adjusted budget for activity A.” The acquisition model maydetermine that the information type for the information request based onone or more keywords in the information request using Natural LanguageProcessing (NLP) algorithms, Natural Language Understanding (NLU)algorithms. The acquisition model may match the one or more keywordspresent in the information request to keywords from the training dataand determine the information type corresponding to an informationrequest in the training data having the matching one or more keywords.In the above example, let us assume that the keyword is “Validate.” Letus assume that the training data may have an informationrequest—“Validate the list of participants.” and the correspondinginformation type—“Judgement.” The acquisition model may determine thatthe information type for the information request in the example is“Judgement.”

After determining the information type, the engagement engine maydetermine a set of human agents for the information request based on theproblem data. The set of human agents may be determined using anengagement model. The engagement model is machine learning based modelwhich is trained using a training data comprising a plurality ofinformation requests, a plurality of lists of one or more human agentscorresponding to the plurality of information requests, problem datacorresponding to each of the plurality of information requests, aplurality of sets of human agents for each information request from theplurality of information requests. It may be noted that the plurality oflists of one or more human agents includes responsibilities andseniority of each human agent from the list of the one or more humanagents. Further, the problem data may include the intermediate stepcorresponding to the information request from the decision-makingprocess and the importance of the intermediate step. In an embodiment,the set of human agents for the information request may comprise one ormore subsets of human agents for one or more intermediate stepsassociated with the information request.

In an embodiment, the engagement model may be trained to calculate aninformation value, a participation value, and a human involvement costfor each human agent from the list of one or more human agentsassociated to the information request. The information value maycorrespond to a merit of the input provided by a human agent via theuser device. The merit may depend on correctness, relevance, anddetails. The participation value may correspond to importance ofinvolvement of a human agent for an information request. The cost ofhuman involvement may be proportional to at least a time of response toan information request, availability, and seniority of a human agent.

The information value may be calculated based on the historic datacomprising past information requests where the human agent was involved,the input provided by the human agent for the past information requests,information used by the collaborative decision-making system for thedecision-making process associated with the information request.

Consider an example, let us assume that a “Human Agent A” was involvedin 8 past information requests. Input provided by “Human Agent A” wasused for 2 past information requests received from the collaborativedecision-making system. The information value for “Human Agent A” may becalculated as 2/8.

Further, the participation value may be calculated based on theintermediate step associated with the information request, theimportance of the intermediate step, responsibilities, and seniority ofthe one or more human agents, and historic data comprising feedbackprovided for past information requests associated to a problem. Thefeedback may be negative or positive with respect to the set of humanagents determined for the past information requests.

Consider an example, let us assume that the list of one or more humanagents comprises of a “Human agent B” and a “Human Agent C.” Let usassume that the “Human Agent B” is CEO of an organization B, and the“Human Agent C” is an intern at the organization B. The informationrequest is “Approve the final decision of terminating one or moreemployees of organization B.” Let us assume that the “Human Agent B” hasprovided feedback for 70 past information requests similar to theinformation request in the example. Let us assume that the feedback wasnegative when the “Human Agent B” was not in the set of human agents forone or more past information requests of the 70 past informationrequests, and vice-versa. Let us assume that “Human Agent C’ hasprovided 0 feedbacks. The participation value of the “Human Agent B”will be higher than the participation value of “Human Agent C.”

The cost of human involvement may be calculated based on historic datacomprising time of response for past information requests, seniority,and availability of the human agent. Consider an example, let us assumethat the list of one or more human agents comprises a “Human Agent D,”and a “Human Agent E.” The “Human Agent D” has been involved in 4 pastinformation requests, and the “Human Agent E” has been involved in 5past information requests. Let us assume that the “Human Agent D” is amanager that works for 3 hours a day, and “Human Agent E” is a managerthat works for 8 hours a day. Let us assume that time of response of the“Human Agent D” for the past information requests is 12 hours, 16 hours,20 hours, and 14 hours. Let us assume that the time of response of the“Human Agent E” for the 5 past information requests is 1 hour, 2 hours,30 minutes, 1 hour, and 1 hour. The cost of human involvement for “HumanAgent D” will be higher than the cost of human involvement of “HumanAgent E.” Considering the same example, if the “Human Agent E” is on aleave for 5 days and the “Human Agent D” is working at the time ofreceiving a new information request, the cost of human involvement forthe “Human Agent D” will be lower than the cost of human involvement forthe “Human Agent E”.

In an embodiment the information value, participation value, and thecost of human involvement may be a value in the range of 1-10. Theinformation value, the participation value, and the cost of humaninvolvement may be calculated using a training data comprising aplurality of information requests, a plurality of lists of one or morehuman agents corresponding to the plurality of information requests,corresponding problem data for the plurality of human requests,information value for each human agent of the plurality of lists of oneor more human agents, participation value for each human agent of theplurality of lists of one or more human agents, and the cost of humaninvolvement for each human agent of the plurality of lists of one ormore human agents.

In an embodiment, the engagement model may calculate a confidence scorefor each human agent based on the information value, the participationvalue, and the cost of human involvement of the human agent. Theconfidence score may be a value in a range of 1-10. Further, theengagement model may determine whether the human agent may be in the setof human agents for an information request based on a thresholdconfidence score. The threshold confidence score may be predefined forthe information request. It may be noted that the confidence score ofthe human agent may be updated continuously after providing an input.

In an embodiment, the engagement model may determine the set of humanagents from the list of one or more human agents for an informationrequest based on the information value, the participation value, and thecost of human involvement. In an embodiment, the information request mayhave a predefined threshold of the information value, the participationvalue, and the cost of human involvement. The set of human agents maycomprise one or more human agents having the information value greaterthan the predefined threshold of information value, the participationvalue greater than the predefined threshold of participation value, andthe cost of human involvement less than the predefined threshold. It maybe noted that the engagement model is continuously trained usingrecursive learning. The engagement model is trained using historic datathat is continuously updated and modified based on the input receivedfrom the set of human agents in real-time.

In an embodiment, the engagement model may extract the historic data ofthe human agents from the one or more user devices 104. The human agentsmay create an account on their respective user devices to enlistthemselves as a part of the list of one or more human agents. Eachiteration of receiving an information request and providing an input forthe information request may be generate and expand the historic data onthe user device or a server connecting the user device to the system.

Further to determining the set of human agents, the engagement enginemay determine a Request Elicitation Type (RET) for the set of humansbased on the information type, and the problem data. The engagementengine may use an elicitation model to determine the RET. It may benoted that the RET may be different for each human agent from the set ofhuman agents, the RET may be different for a human agent for differentinformation requests. The RET may correspond to how an informationrequest may be framed for a human agent. In other words, the RET maycorrespond to a method used to communicate the information request tothe human agent. RET may be at least one of a comparison type, an optiontype, an audio type, a video type, a brief answer question type, a longanswer question type, and the like.

In an embodiment, the engagement engine may generate a human agentprofile for the human agent based on the historic data comprising pastinformation requests answered by the human agent, the input provided forthe past information requests, the RET of the past information requests,responsibilities, and seniority of the human agent. Further, theengagement engine may generate RET profiles for each RET using externaldata sources such as internet, research papers, surveys, and the like.The RET profiles may indicate suitability of the RET for different humanagent profiles, different information requests, different informationtypes, different problems associated with the information requests.

The elicitation model may use a machine learning algorithm trained usingtraining data comprising a plurality of information requests,corresponding problem data for the plurality of information requests,corresponding sets of human agents for the plurality of informationrequests, human agent profiles of each human agent from the sets ofhuman agents, RET profiles, RET for each human agent for thecorresponding information requests. The elicitation model may be trainedto produce an RET for each human agent in a set of human agents for aninformation request as an output for an input of the informationrequest, the set of human agents, the problem data. The elicitationmodel may compare the human profile of a human agent in the set of humanagents with the human agent profiles of human agents in the trainingdata and select the RET of the human agent in the training data having asimilar human profile to the human agent in the set of human agents forthe information request.

In an embodiment, the elicitation model may match human agent profilesof each human agent from the set of human agents with the RET profilesto determine the most compatible RET for each human agent.

Further to determining the RET for each human agent of the set of humanagents, the engagement engine may receive input required for theinformation request from the set of human agents. It may be noted thatthe input required for the information request may be different for eachhuman agent from the set of human agents. The engagement engine maydisplay the information request to the set of human agents based on theRET and the information types. The human agents may use a user device toprovide the input using a graphical user interface. The input may be anaudio response, a video response, a test response, a selection, agraphical response, and a feedback. The feedback may correspond torelevance of the information request for the human agent. The feedbackmay correspond to at least relevance of the information request for thehuman agent, relevance of the RET for the human agent, correctness ofthe information type for the information request, appropriateness of thehuman agents in the set of human agents. The feedback may either bepositive or negative. In an embodiment, failure to receive the inputfrom a human agent may be considered as negative feedback.

Upon receiving the input from the set of human agents, the engagementengine may retrain the engagement model and the elicitation model basedon the received information. The engagement engine may verify the use ofthe received information with the CDS. In case the input is not used bythe CDS, the training data for the engagement model and the elicitationmodel may be changed for similar types of information requests such thatduring the next iteration of a similar information request, the set ofhuman agents, the RET and the information type do not match the set ofhuman agents, the RET and the information type used for receiving theinput that was not used by the CDS. In an embodiment, upon receivingnegative feedback from a human agent with respect to the RET for aninformation request, the engagement engine may modify the training datafor the elicitation model to avoid negative feedback in futureiterations.

Finally, the engagement engine may use the input received from the setof human agents to continuously enhance the decision-making data. Theenhancement of the decision-making data may improve the efficiency andaccuracy of the system for engagement of human agents with acollaborative decision-making system. Each iteration of the engagementengine may add new data to the historic data of information requests.

Consider an example, a minister of foreign relations of Country A isusing a collaborative decision-making system. The minister needs adecision for a problem, let us assume that the problem is “Improvingrelations with Country B.” In case of such problems, the collaborativedecision-making system may seek at least new information, and approval,from one or more human agents, for one or more intermediate steps in thedecision-making process of the CDS using the engagement engine. Let usassume that the CDS seeks an approval on a list of ministers from theCountry B being invited to a gathering. The information request in caseof the example is “Approval for List of Ministers of Country B invitedto a gathering.” The foreign relations department of a government is alarge body with many members. The system may extract the problem datacomprising the list of members of the foreign relations department ofCountry A and Country B. The list may also include the responsibilitiesand seniority of the members. Further, the system may select theinformation type as “Judgement” since the information request requiresan approval. The information request may also need involvement of one ormore human agents from the foreign relations department of Country Balong with appropriate set of human agents from the foreign relationsdepartment of Country A. The system may select one or more ministers forthe set of human agents from Country A without missing any importantministers that may object to not being involved in the process. Further,the system may select one or more ministers for the set of human agentsfrom Country B to approve the list of ministers attending the gatheringfrom their side. The system ensures involvement of the ministers thatare necessary for the information requests based on theirresponsibilities, information value, participation value, and cost ofhuman involvement. Further, the system may determine the RET for eachminister part of the set of human agents based on at least the languagespoken by them, the device they use, any disabilities they may have, andfeedback provided for RET in any past information request. Finally, theministers that are a part of the set of human agents may enter theirselections of the list of ministers of Country B invited to thegathering.

In an embodiment, the set of human agents may have one or more subsetsof human agents. Considering the above example, ministers of country Athat are part of the set of human agents may be subset 1 of human agentsand ministers of country B that are part of the set of human agents maybe subset 2 of human agents. Further, the information request may bedifferent for both the subsets when they are involved at differentintermediate steps of the decision-making process of the CDS. Let usassume that subset 1 is involved for approving the ministers of countryB selected by the CDS first in intermediate step 1 and subset 2 isinvolved in confirming availability of the selected ministers of countryB and modifying the list in intermediate step 2. The information requestfor subset 1 will be “Approve the selected ministers of country B to beinvited”, and the information request for subset 2 will be “Checkavailability and modify the list of ministers”.

Referring now to FIG. 2 , an example of a communication between theCollaborative Decision-making System (CDS) 102 and a human agent 202using the engagement engine is illustrated. An information requestgenerated by the CDS may be presented to the human agent 202 using theengagement engine. Further, the input received from the human agent forthe information request may be received by the engagement engine. Theinput may be used by the CDS 102 since the engagement engine is a partof the CDS 102.

In an alternate embodiment, the engagement engine may be a stand-alonesystem to determine a set of human agents from the list of human agentsto be involved in a decision-making process for a problem. Further theengagement engine may be used to receive input from the determined setof human agents. The input may be helpful in the decision-making processfor the problem.

Further to receiving the input from the set of human agents, the systemmay filter the new information based on merit. The merit may becalculated by using Artificial Intelligence (AI) techniques,Mathematical Modelling, Markov Decision Modelling, and Deep Learning(DL) algorithms on factors like human bias, historic user judgements,consequences, and credibility of the external data source.

After modifying the decision space, the system 102 may generate adecision knowledge graph depicting modifications in the decision space.The decision knowledge graph may comprise data related to themodifications in the decision space. The decision knowledge graph may bea semantic knowledge graph comprising a list of new decision optionsadded in the decision space, list of decision options removed from thedecision space, a reason for the addition and the removal of thedecision options from the decision space, and the intermediate step whenthe decision space got modified. The reason for the addition or theremoval of the decision options may be caused due to an uncertain eventor the new information.

It may be noted that the system uses a unique shared semantic graph,referred to as the decision knowledge graph, depicting the evolution ofthe decision-space during the act of decision-making. The building ofthe decision knowledge graph may be completed when the decisionknowledge graph has enough knowledge to form a decision-implementingintention. The completed decision knowledge graph can be simply queriedto retrieve information such as:

Information content of the decision; (What must guide the decisionimplementing action?)

The trigger information for the decision implementing information;

Explanation behind what led to commitment to the decision;

Why and when another alternative got eliminated from consideration?

The quantified value and risk associated with decision;

Under what circumstances, the decision will cease to be the preferredchoice;

Further to generating the decision space, the system 102 may update thedecision space and the decision knowledge graph. The decision space maybe updated post validation of the decision knowledge graph by the one ormore human agents. In an embodiment, the one or more human agents mayprovide feedback on the decision space. In another embodiment, thesystem may train based on the feedback provided by the one or moreinput. It may be noted that system continuously learns from the one ormore human agents in order to improve the efficiency of the system. Inan embodiment, when the one or more human agents reject themodifications in the decision space depicted in the decision knowledgegraph the system may again modify the decision space. Further, thesystem may also reconsider the uncertain events.

Further to updating, the system 102 may select the decision based on theupdated decision knowledge graph and the updated decision space. Thefinal decision may be selected post approval of the one or more humanagents. In an embodiment, the decision may be selected by a process ofeliminating the one or more decision options from the decision space. Itmay be noted that the one or more decision options that fails to matchthe criteria or the goal or the constraints may be eliminated.

In an embodiment, the system 102 may reinitiate the collaborativedecision making when no decision satisfies the goals, constraints andsuccess measures or does not receive approval of the human agents. Itmay be noted that the decision may be selected from the updated decisionspace.

Consider an example, let us assume that a user is a representative of anorganisation. The system may receive a query corresponding to a problemfrom the user. Let us assume that the problem is “hiring Data Analysts.”The system may define the problem using metadata. The metadata maycomprise a goal, constraints, success measures and a list of humanagents. In the above example, let us assume that the goal is “hiringeight Data Analysts.” Further, the constraints may be “the wage offeredshould not exceed $250,000,” “the Data Analysts must join on 1st ofJanuary 2022,” “The Data Analyst must have an Engineering degree inComputer Science,” and “the Data Analysts must sign a contract of twoyears”. Let us assume that the success measures are “the Data Analystsjoin on 1^(st) of January 2022,” “the Data Analysts work until 1^(st) ofJanuary 2024,” “eight Data Analysts must be hired,” “the Data Analystsmust be from an Engineering background,” and “the Data Analysts must bechosen from a list of applicants.” In the example, the one or more humanagents involved may be a head of human resources, an associate humanresources manager, a project manager, and a project leader. The systemmay calculate one or more intermediate steps to select eight DataAnalysts out of the list of the applicants that fulfil the goals,constraints, and success measures. Further, the system may establish adecision-making flow for the one or more intermediate steps by arrangingthem in a sequence. Let us assume the decision-making flow for the oneor more intermediate steps for the given example is

-   -   1. Receive the list of applicants from the associate human        resources.    -   2. Ask the head of the human resources to confirm the list of        the applicants and receive amendments, if any.    -   3. Filter the applicants from the list of the applicants and        share the filtered list of the applicants with the project        manager.    -   4. Receive amendments in the filtered list of the applicants        from the project manager, if any and prepare a final list of        applicants.    -   5. Share the final list of the applicants with the project        leader for assessment.    -   6. Receive results of the assessment from the project leader.    -   7. Add new applicants to the final list of the applicants if any        applicant is eliminated in the assessment process and share the        new applicants with the project leader for assessment.    -   8. Reiterate steps 5,6 and 7 until eight Data Analysts are        finalised.    -   9. Share the list of eight Data Analysts to hire with Associate        Human Resources to validate.

A decision space comprising a plurality of decision options may begenerated. The plurality of decision options may comprise the list ofthe applicants. The decision space may be modified at every step basedon the inputs from the human agents and analysis. Considering the aboveexample, the system may identify a risk that the list of applicants isnot enough to select eight Data Analysts. The system may calculate theconsequence of the risk that sufficient interns are not found out of thelist of applicants. In an embodiment, the consequences may impact thegrowth of the organization, or efficiency of the employees, or alike. Inorder to reduce the impact of the consequence, the system may determinea risk informed strategy. In an embodiment, the system may providepredictive suggestion to the one or more human agents to obtain moreapplicants for the list of applicants. The system may nudge theassociate human resources manager to obtain more applicants. The list ofapplicants may be modified based on the system's suggestion and theinput from the associate human resources manager. The associate humanresources manager may also choose to ignore the system's suggestionafter reviewing the consequences and risk. Let us assume that thesuggestion was ignored. The system may continuously monitor the activityof the list of the applicants on social media and check for any emailsfrom any applicants for any new information. Consider an example, thesystem may identify that an applicant from the list of applicants joineda competitor firm for the period 1 Jan.-1 Jul. 2022 from a database ofprofessional workers or social media. This may be considered as anuncertain event which may lead to removal of the applicant from thedecision space. A decision knowledge graph may be generated that depictsall the modifications made in the decision space.

For the above example, let us assume that the associate human resourcesmanager shares a list of 80 applicants. The head of human resourcesamends the list and removes 10 applicants. The decision knowledge graphdepicts this modification and shows that the list was modified at Step 2by the head of human resources and 10 applicants were removed from thedecision space. Any other modifications made during the decision-makingflow will also be recorded in a similar manner. When the system reachesa decision, the stakeholders may validate the decision knowledge graphand check all the modifications and provide feedback. The system mayselect a final decision. For this example, the final decision may be thelist of eight Data Analysts to be hired.

Consider another example, let us assume that a user is representative ofan organization. The system may receive a query corresponding to aproblem the user is confronting. Let us assume that the problem is aboutdeciding on a concept for a new product (for example, a flying car) thatmeets the needs of an emerging marketplace better than competingproducts. The system may define the problem using metadata. The metadatamay comprise a goal, constraints, success measures, and a list of humanagents. Let us assume that the goal is to develop an engineered product(for example, a flying car) conforming to the decided concept. Theproduct concept is defined as a combination of capabilities (forexample-seating capacity, engine type, flying height, etc.) whereas eachcapability has multiple options (for example, engine type may begasoline, hybrid, plug-in hybrid, or fully electric motor).

Further the constraints may be that the product concept must befinalized within 6 months, and the concept development cost must notexceed $2 million. Let us assume that the success measures are “theconcept must be feasible to manufacture,” “the total per unitmanufacturing cost should not exceed $15,000”, “the product should havezero defects”, and “the product must be ready to hit the market in 2years after concept development”.

In the example, the one or more human agents may be market researcherswho represent the perceived needs of the marketplace, product manager,technology subject matter experts and chief financial officer.

Further, the system may calculate one or more intermediate steps toreach the decision that fulfil the goals, constraints, and successmeasures, and for selecting the best concept. Further, the system mayestablish a decision-making flow for the one or more intermediate stepsby arranging them in an optimal sequence.

Let us assume that the decision-making flow for the one or moreintermediate steps for the given example is:

-   -   1. Receive the list of requirements from market researchers.    -   2. Develop a list of distinct product concept capabilities to        meet the requirements.    -   3. For each capability, identify a list of discrete capability        concept variations as options for achieving that capability.        This may involve ideas from technology subject-matter experts        and/or searching an existing knowledge base for the product        domain.    -   4. Assess each capability option in terms of design and        manufacturing feasibility within the given constraints. This        assessment may involve judgements from technology subject-matter        experts, and artificial intelligence techniques to infer        infeasibility.    -   5. Eliminate capability options which are infeasible.    -   6. Generate all possible product concepts by combining a        feasible option from each of the identified capabilities. This        may require describing product concepts by one or more decision        space membership rules as explicit enumeration of all concepts        may be quite complex to handle.    -   7. Not all concepts in step 6 may be valid because of        dependencies among options of different capabilities. That is        certain options may not work together in a concept whereas        certain option combinations may always have to work together.        Identify such dependencies by consulting subject-matter experts        and leveraging prior product domain knowledge.    -   8. Efficiently eliminate invalid product concepts from further        consideration at the right moment.    -   9. If the list of remaining product concepts is small or the        time and budget for concept development is exhausted, go to step        14 otherwise go to step 10.    -   10. Select a subset of product concepts to be evaluated further.        This is accomplished by using a criteria which is likely to lead        to faster and cheaper convergence to the final product concept.    -   11. Evaluate each product concept in terms of success criteria        attributes and relative preference attributes.    -   12. Eliminate all product concepts from further consideration        which are inferior to some other remaining product concept as        informed by step 11 and are likely to remain so in spite of new        information in the target time frame for decision making.    -   13. Re-iterate steps 9-12.    -   14. Compare all remaining product concepts to select the most        desirable product concept.

Further to establishing decision making flow, a decision spacecomprising a plurality of decision options may be generated. Theplurality of decision options may comprise the list of all productconcepts which may be explicitly recorded or described by the one ormore decision space membership rules. Using the one or more decisionspace membership rules, one can infer whether or not an arbitraryproduct concept belongs to the decision space. The decision space may bemodified at each step based on the inputs from the one or more humanagents and analysis. Considering the above example, the system mayidentify a risk that the identified product concepts may not be adequateto explore newer and novel capabilities. The system may calculate theconsequences of the risk to the organization. In an embodiment, theconsequences may impact the competitiveness of the manufactured productconforming to the selected concept, thereby, reducing the market shareof the organization. In order to alleviate or minimize unintendedconsequential risk, the system may determine a risk informed strategy.

The system may engage the engagement engine to receive the input fromthe one or more human agents. It may be noted that the one or more humanagents or a set of human agents are determined from a list of one ormore human agents depending on the problem. The set of human agents aredetermined by using an engagement model. Further, the engagement enginemay nudge the set of human agents to receive the input. The input may bereceived in multiple types such as facts, opinions, and judgements. Theengagement engine automatically determines the information type based onthe problem or the step for which a decision or input is required. Inthe example, the engagement engine may nudge the set of human agents tovalidate “the product design of the flying car.” It must be noted thatthe system may nudge a different set of human agents at a differentstage of deciding on a concept for a new product. The different stagesmay comprise, but not limited to, idea generation, idea screening,concept development, testing, market strategy, product development,deployment and a like. Further, the engagement engine may involve theset of human agents such as decision maker, stake-holder, expert, andalike. In the example and not by way of any limitation, the engagementengine may determine that it is not suitable to involve decision makersat the product ideation stage because the cost involved is very high. Inanother scenario, when the decision is complex and the stakes are high,the engagement engine may determine that the involvement of decisionmakers and stake holders is required.

In an embodiment, the engagement engine may provide a predictivesuggestion to the one or more human agents to generate more productconcepts as an information request. The system may nudge the productmanager and the technology subject matter experts to consider newercapabilities in arriving at an expanded list of product concepts. Thelist of product concepts may be modified based on the system'ssuggestion and the input from technology subject matter experts (humanagents). The product manager may also choose to ignore the system'ssuggestion after reviewing the consequences and risk. Let us assume thatthe suggestion was ignored. The system may continuously monitordevelopments in the marketplace which may influence the currentperception of the risk. For example, newer experiences with flying carconcepts may indicate that flying below or above a certain height hasunacceptable safety risks. It may also be the case that a new regulationis passed which may impose a restriction on the flying height. This willbe considered as an uncertain event which may invalidate an existingcapability option and suggest one or more new capability options therebyimpacting the decision space (product concept space). A decisionknowledge graph may be generated that depicts all the modifications madein the decision space along with the rationale for modifications.

Further, the decision knowledge graph and the decision space may beupdated post validation of one or more human agents. Finally, a decisionis selected based on the updated decision knowledge graph. It may benoted that the decision is selected after consideration of all theuncertain events (e.g. restriction on the flying height). The finaldecision may be a flying car that meets the expectation of the user.When the user is not satisfied with the final decision the process isreiterated.

In an embodiment, the system 102 may also be referred as a decisionintelligence system. It may be noted that the decision intelligencesystem is an adaptive and intelligent automated system that collaborateswith human decision-maker (decision-making group or human agents) tohelp transition them from decision-making intention mental state todecision-implementing intention state.

Referring now to FIG. 3 , a method 300 for engagement of human agentswith a collaborative decision-making system is shown, in accordance withan embodiment of the present subject matter. The method 300 may bedescribed in the general context of computer executable instructions.Generally, computer executable instructions can include routines,programs, objects, components, data structures, procedures, modules,functions, etc., that perform particular functions or implementparticular abstract data types.

The order in which the method 300 is described is not intended to beconstrued as a limitation, and any number of the described method blockscan be combined in any order to implement the method 200 or alternatemethods for engagement of human agents for decision-making in adynamically changing environment. Additionally, individual blocks may bedeleted from the method 200 without departing from the scope of thesubject matter described herein. Furthermore, the method 200 forengagement of human agents for decision-making in a dynamically changingenvironment can be implemented in any suitable hardware, software,firmware, or combination thereof. However, for ease of explanation, inthe embodiments described below the method 200 may be considered to beimplemented in the above-described system 102.

At block 302, an information request relating to a problem may bereceived.

At block 304, problem data comprising metadata associated to theproblem, and decision-making data may be received.

At block 306, an information type for the information request may bedetermined based on the problem data using an acquisition model.

At block 308, a set of human agents from a list of one or more humanagents may be determined for the information request based on theproblem data using an engagement model.

At block 310, a request elicitation type may be determined for the setof human agents based on the problem data and the information type usingan elicitation model.

At block 312, input for the information request may be received from theset of human agents based on the request elicitation type and theinformation type.

At block 314, the engagement model and the elicitation model may beretrained based on the input received from the set of human agents usingrecursive learning techniques.

At block 316, the decision-making data may be continuously enhancedbased on the input received, the request elicitation type, and theinformation type.

Referring now to FIG. 4 , a method 400 for collaborative decision makingin dynamically changing environment is shown, in accordance with anembodiment of the present subject matter. The method 200 may bedescribed in the general context of computer executable instructions.Generally, computer executable instructions can include routines,programs, objects, components, data structures, procedures, modules,functions, etc., that perform particular functions or implementparticular abstract data types.

The order in which the method 400 is described is not intended to beconstrued as a limitation, and any number of the described method blockscan be combined in any order to implement the method 400 or alternatemethods for collaborative decision making in dynamically changingenvironment. Additionally, individual blocks may be deleted from themethod 400 without departing from the scope of the subject matterdescribed herein. Furthermore, the method 400 for collaborative decisionmaking in dynamically changing environment can be implemented in anysuitable hardware, software, firmware, or combination thereof. However,for ease of explanation, in the embodiments described below the method400 may be considered to be implemented in the above-described system302.

At block 402, a query corresponding to a problem for which a decision isrequired may be received from a user.

At block 404, one or more intermediate steps required to reach adecision may be calculated based on metadata. The one or moreintermediate steps may be calculated using reinforcement learning, deeplearning, and artificial intelligence algorithms. The metadata may beassociated to the problem.

At block 406, a decision-making flow for the one or more intermediatesteps required to reach the decision may be established. Thedecision-making flow may correspond to an order in which the one or moreintermediate steps must be taken. The order may be determined based onan importance and a severity of the one or more intermediate steps usingreinforcement learning and artificial intelligence techniques.

At block 408, a decision space comprising one or more decision optionsmay be generated. The decision space may be calculated based on thedecision-making flow using artificial intelligence and deep learning.

At block 410, the decision space may be modified based on one or moreuncertain events. The one or more uncertain events may create anuncertain impact on the decision-making flow. The one or more uncertainevents may be determined based on new information.

At block 412, a decision knowledge graph depicting modifications in thedecision space may be generated.

At block 414, the decision space and the decision knowledge graph may beupdated. The decision space may be updated post validation of thedecision knowledge graph by the one or more human agents.

At block 416, the decision may be selected based on the updated decisionknowledge graph and the updated decision space. The final decision maybe selected post approval of the one or more human agents.

FIG. 5 illustrates an example artificial neural network (“ANN”) 500 ofthe deep learning algorithms used to train the engagement model, and theelicitation model. In particular embodiments, an ANN may refer to acomputational model comprising one or more nodes. Example ANN 500 maycomprise an input layer 510, hidden layers 520, 530, 560, and an outputlayer 550. Each layer of the ANN 500 may comprise one or more nodes,such as a node 505 or a node 515. In particular embodiments, each nodeof an ANN may be connected to another node of the ANN. As an example,and not by way of limitation, each node of the input layer 510 may beconnected to one of more nodes of the hidden layer 520. In particularembodiments, one or more nodes may be a bias node (e.g., a node in alayer that is not connected to and does not receive input from any nodein a previous layer). In particular embodiments, each node in each layermay be connected to one or more nodes of a previous or subsequent layer.Although FIG. 5 depicts a particular ANN with a particular number oflayers, a particular number of nodes, and particular connections betweennodes, this disclosure contemplates any suitable ANN with any suitablenumber of layers, any suitable number of nodes, and any suitableconnections between nodes. As an example, and not by way of limitation,although FIG. 5 depicts a connection between each node of the inputlayer 510 and each node of the hidden layer 520, one or more nodes ofthe input layer 510 may not be connected to one or more nodes of thehidden layer 520.

In particular embodiments, an ANN may be a feedforward ANN (e.g., an ANNwith no cycles or loops where communication between nodes flows in onedirection beginning with the input layer and proceeding to successivelayers). As an example, and not by way of limitation, the input to eachnode of the hidden layer 520 may comprise the output of one or morenodes of the input layer 510. As another example and not by way oflimitation, the input to each node of the output layer 550 may comprisethe output of one or more nodes of the hidden layer 560. In particularembodiments, an ANN may be a deep neural network (e.g., a neural networkcomprising at least two hidden layers). In particular embodiments, anANN may be a deep residual network. A deep residual network may be afeedforward ANN comprising hidden layers organized into residual blocks.The input into each residual block after the first residual block may bea function of the output of the previous residual block and the input ofthe previous residual block. As an example, and not by way oflimitation, the input into residual block N may be F(x)+x, where F(x)may be the output of residual block N−1, x may be the input intoresidual block N−1. Although this disclosure describes a particular ANN,this disclosure contemplates any suitable ANN.

In particular embodiments, an activation function may correspond to eachnode of an ANN. An activation function of a node may define the outputof a node for a given input. In particular embodiments, an input to anode may comprise a set of inputs. As an example, and not by way oflimitation, an activation function may be an identity function, a binarystep function, a logistic function, or any other suitable function.

In particular embodiments, the input of an activation functioncorresponding to a node may be weighted. Each node may generate outputusing a corresponding activation function based on weighted inputs. Inparticular embodiments, each connection between nodes may be associatedwith a weight. As an example, and not by way of limitation, a connection525 between the node 505 and the node 515 may have a weightingcoefficient of 0.4, which may indicate that 0.4 multiplied by the outputof the node 505 is used as an input to the node 515. In particularembodiments, the input to nodes of the input layer may be based on avector representing an object. Although this disclosure describesparticular inputs to and outputs of nodes, this disclosure contemplatesany suitable inputs to and outputs of nodes. Moreover, although thisdisclosure may describe particular connections and weights betweennodes, this disclosure contemplates any suitable connections and weightsbetween nodes.

In particular embodiments, the ANN may be trained using training data.As an example, and not by way of limitation, training data may compriseinputs to the ANN 500 and an expected output. As another example and notby way of limitation, training data may comprise vectors eachrepresenting a training object and an expected label for each trainingobject. In particular embodiments, training the ANN may comprisemodifying the weights associated with the connections between nodes ofthe ANN by optimizing an objective function. As an example, and not byway of limitation, a training method may be used (e.g., the conjugategradient method, the gradient descent method, the stochastic gradientdescent) to backpropagate the sum-of-squares error measured as adistances between each vector representing a training object (e.g.,using a cost function that minimizes the sum-of-squares error). Inparticular embodiments, the ANN may be trained using a dropouttechnique. As an example, and not by way of limitation, one or morenodes may be temporarily omitted (e.g., receive no input and generate nooutput) while training. For each training object, one or more nodes ofthe ANN may have some probability of being omitted. The nodes that areomitted for a particular training object may be different than the nodesomitted for other training objects (e.g., the nodes may be temporarilyomitted on an object-by-object basis). Although this disclosuredescribes training the ANN in a particular manner, this disclosurecontemplates training the ANN in any suitable manner.

Exemplary embodiments discussed above may provide certain advantages.Though not required to practice aspects of the disclosure, theseadvantages may include those provided by the following features.

In some embodiments, the system may improve efficiency and speed of thedecision-making.

In some embodiments, the system provides better decisions due to theinvolvement of human agents in the decision-making process.

In some embodiments, the system may help in reduction of manpower byautomating managerial tasks, making decisions on behalf of the humans.

In some embodiments, the system may help in streamlining conversationsbetween the one or more human agents involved in the decision-makingprocess.

In some embodiments, the system alerts a user for an uncertain eventimpacting the decision.

In some embodiments, the system may augment the decision when there isan uncertain event creating an uncertain impact on the decision.

In some embodiments, the system may help to reduce the impact of the oneor more uncertain events.

In some embodiments, the engagement engine may improve efficiency andspeed of the decision-making.

In some embodiments, the system provides better decisions due to betterinvolvement of one or more human agents in the decision-making process,wherein the one or more human agents provide information to theengagement engine to take part in the decision making process.

In some embodiments, the engagement engine may help in reduction ofmanpower by selecting appropriate number of human agents involved in thedecision-making process.

In some embodiments, the engagement engine may help in streamliningconversations between the one or more human agents involved in thedecision-making process and the collaborative decision-making system.

Although implementations for methods and the systems for engagement ofhuman agents for decision-making in a dynamically changing environmenthave been described in a language specific to structural features and/ormethods, it is to be understood that the appended claims are notnecessarily limited to the specific features or methods described.Rather, the specific features and methods are disclosed as examples ofimplementations for engagement of human agents for decision-making in adynamically changing environment.

What is claimed is:
 1. A system for engagement of human agents fordecision-making in a dynamically changing environment, the systemcomprises: a memory 112; and a processor 108 coupled to the memory,wherein the processor is configured to execute instructions stored inthe memory for: receiving an information request relating to a problemrequiring a decision; receiving problem data comprising metadataassociated to the problem, and decision-making data; determining aninformation type, based on the problem data, using an acquisition model,wherein the information type is at least a fact, an opinion, and ajudgement; determining a set of human agents from a list of one or morehuman agents for the information request based on the problem data,wherein the set of human agents are determined by using an engagementmodel; determining a Request Elicitation Type (RET) for the set of humanagents based on the problem data and the information type using anelicitation model; receiving an input from the set of human agents forthe information request based on the information type, and the requestelicitation type; retraining the engagement model and the elicitationmodel based on the received input using recursive learning techniques;and continuously enhancing the decision-making data based on thereceived input, the determined request elicitation type, and thedetermined information type.
 2. The system in claim 1, wherein theinformation request is received from a collaborative decision-makingsystem, and wherein the collaborative decision-making system is used bya user to obtain a decision corresponding to the problem.
 3. The systemin claim 1, wherein the received input is at least a text response, avisual response, an audio response, a video response, and a feedbackbased on the information type.
 4. The system in claim 3, wherein thefeedback is either negative or positive, and wherein the acquisitionmodel is trained based on the feedback using recursive learningtechniques.
 5. The system in claim 1, wherein the metadata comprises agoal, constraints, success measures, a list of the one or more humanagents involved in a decision-making process, and historic datacomprising input received from the one or more human agents, a successratio of the input received from the one or more human agents, andwherein the list of the one or more human agents definesresponsibilities and seniority of each of the one or more human agents.6. The system in claim 1, wherein the decision-making data comprises oneor more intermediate steps, importance of the one or more intermediatesteps, a decision-making flow, and historic data comprising pastinformation requests received.
 7. The system in claim 1, whereinselecting a set of human agents further comprises calculating aparticipation value, an information value and a human involvement costbased on the metadata using the engagement model.
 8. The system in claim1, wherein the Request Elicitation Type (RET) corresponds to framing ofthe information request for a human agent, and wherein the determinedrequest elicitation type is used to receive the input from thedetermined set of human agents.
 9. The system in claim 1, whereindetermining the Request Elicitation Type (RET) further comprisesgenerating a human agent profile, accessing one or more requestelicitation type profiles, matching the human agent profile with the oneor more request elicitation type profiles using the elicitation model.10. The system in claim 6, wherein at least an intermediate step of theone or more intermediate step has a corresponding information request,and wherein the set of human agents comprises one or more subsets ofhuman agents for the one or more intermediate steps.
 11. The system inclaim 5, wherein the goal represents an objective or final expectationof the user, and the constraints represent limitations of resourcesavailable to achieve the goal and the success measures representparameters utilized for confirmation of achievement of the goal.
 12. Thesystem in claim 1, wherein the engagement model is a machine learningmodel continuously trained using inputs provided by the set of humanagents.
 13. The system in claim 1, wherein the elicitation model is amachine learning model continuously trained using inputs provided by theset of human agents.
 14. A method for engagement of human agents fordecision-making in a dynamically changing environment, the methodcomprises: receiving, by a processor, an information request relating toa problem requiring a decision; receiving, by the processor, problemdata comprising metadata associated to the problem, and decision-makingdata; determining, by the processor, an information type, based on theproblem data, using an acquisition model, wherein the information typeis at least a fact, an opinion, and a judgement; determining, by theprocessor, a set of human agents from a list of one or more human agentsfor the information request based on the problem data, wherein the setof human agents are determined by using an engagement model;determining, by the processor, a Request Elicitation Type (RET) for theset of human agents based on the problem data and the information typeusing an elicitation model; receiving, by the processor, an input fromthe set of human agents for the information request based on theinformation type, and the request elicitation type; retraining, by theprocessor, the engagement model, and the elicitation model based on thereceived input using recursive learning techniques; and continuouslyenhancing, by the processor, the decision-making data based on thereceived input, the determined request elicitation type, and thedetermined information type.
 15. The method in claim 14, wherein theinformation request is received from a collaborative decision-makingsystem, and wherein the collaborative decision-making system is used bya user to obtain a decision corresponding to the problem.
 16. The methodin claim 14, wherein the received input is at least a text response, avisual response, an audio response, a video response, and a feedbackbased on the information type.
 17. The method in claim 16, wherein thefeedback is either negative or positive, and wherein the acquisitionmodel is trained based on the feedback using recursive learningtechniques.
 18. The method in claim 14, wherein the metadata comprises agoal, constraints, success measures, a list of the one or more humanagents involved in a decision-making process, and historic datacomprising input received from the one or more human agents, a successratio of the input received from the one or more human agents, andwherein the list of the one or more human agents definesresponsibilities and seniority of each of the one or more human agents.19. The method in claim 14, wherein the decision-making data comprisesone or more intermediate steps, importance of the one or moreintermediate steps, a decision-making flow, and historic data comprisingpast information requests received.
 20. The method in claim 14, whereinselecting a set of human agents further comprises calculating aparticipation value, an information value and a human involvement costbased on the metadata using the engagement model.
 21. The method inclaim 14, wherein the Request Elicitation Type (RET) corresponds toframing of the information request for a human agent, and wherein thedetermined request elicitation type is used to receive the input fromthe determined set of human agents.
 22. The method in claim 14, whereindetermining the Request Elicitation Type (RET) further comprisesgenerating a human agent profile, accessing one or more requestelicitation type profiles, matching the human agent profile with the oneor more request elicitation type profiles using the elicitation model.23. The method in claim 19, wherein at least an intermediate step of theone or more intermediate step has a corresponding information request,and wherein the set of human agents comprises one or more subsets ofhuman agents for the one or more intermediate steps.
 24. The method inclaim 18, wherein the goal represents an objective or final expectationof the user, and the constraints represent limitations of resourcesavailable to achieve the goal and the success measures representparameters utilized for confirmation of achievement of the goal.
 25. Themethod in claim 14, wherein the engagement model is a machine learningmodel continuously trained using inputs provided by the set of humanagents.
 26. The method in claim 14, wherein the elicitation model is amachine learning model continuously trained using inputs provided by theset of human agents.
 27. A non-transitory computer program producthaving embodied thereon a computer program for engagement of humanagents for decision-making in a dynamically changing environment, thecomputer program product storing instructions for: receiving aninformation request relating to a problem requiring a decision;receiving problem data comprising metadata associated to the problem,and decision-making data; determining an information type, based on theproblem data, using an acquisition model, wherein the information typeis at least a fact, an opinion, and a judgement; determining a set ofhuman agents from a list of one or more human agents for the informationrequest based on the problem data, wherein the set of human agents aredetermined by using an engagement model; determining a RequestElicitation Type (RET) for the set of human agents based on the problemdata and the information type using an elicitation model; receiving aninput from the set of human agents for the information request based onthe information type, and the request elicitation type; retraining theengagement model, and the elicitation model based on the received inputusing recursive learning techniques; and continuously enhancing thedecision-making data based on the received input, the determined requestelicitation type, and the determined information type.